DXL Script Performance

I have a script that is a modification of the commas.dxl import. The script uses data in each of the line of a CSV file to add, change, or delete objects and in which module. If the module doesn't exist it creates it. I have noticed the longer the script runs the slower it is processing the lines in the CSV. At the begining it is processing around 60 line a second and by then end of 3000 lines it has slowed to around 1 line a second. Also during the slow down there will be points when the script will suddenly speed up for a little while and then slow back down. I have watched the cpu usage while the script is running and as it slows down the cpu usage goes up all the way until it his the max. I have also watched the cpu usage on the server and it goes down as the scripts slows.

I had thought that maybe the problem was that I was using too many strings in the script cause the string table to get too large. But I changed the strings to buffers and this made the performace worse.

Does anyone have any ideas on what might be causing this slow down? Could there be some other processing that DOORS is doing that would cause the script to slow? Any help would be appreciated.

Thanks,
Lee
Lee_Bost - Thu Feb 19 13:03:17 EST 2009

Re: DXL Script Performance
Doug.Zawacki - Thu Feb 19 13:55:26 EST 2009

The best way to give advice on this is if you can provide the script itself. I realize sometimes this is not feasible so if you cannot then we may have to talk more in general terms.

Because you eliminated the string concatenation problem and started using buffers with slower performance, I might think that you are creating these buffers in a called function. I have seen buffer creation in functions called many times cause a degradation of performance. If it's at all possible I would suggest trying to have a global buffer.

Sorry, I can't comment much more due to lack of understanding what is going on behind the scene.

Re: DXL Script Performance
Lee_Bost - Thu Feb 19 15:18:32 EST 2009

Doug.Zawacki - Thu Feb 19 13:55:26 EST 2009
The best way to give advice on this is if you can provide the script itself. I realize sometimes this is not feasible so if you cannot then we may have to talk more in general terms.

Because you eliminated the string concatenation problem and started using buffers with slower performance, I might think that you are creating these buffers in a called function. I have seen buffer creation in functions called many times cause a degradation of performance. If it's at all possible I would suggest trying to have a global buffer.

Sorry, I can't comment much more due to lack of understanding what is going on behind the scene.

I understand what you mean. I just tried a simply change using a global buffer still caused an even greater slow down. I am attaching the script. Let me know if you have any suggestions.
Attachments

attachment_14219389_import.dxl

Re: DXL Script Performance
Doug.Zawacki - Thu Feb 19 16:35:08 EST 2009

Lee_Bost - Thu Feb 19 15:18:32 EST 2009
I understand what you mean. I just tried a simply change using a global buffer still caused an even greater slow down. I am attaching the script. Let me know if you have any suggestions.

I will take a look at the script. But I do have another question. When you are watching the CPU, how much memory are you using up? How much memory is on the workstation the script is running? Are you swapping memory out because of low memory? All these things can contribute to the issue. Of course, DXL Slowing down the longer it runs usually means it's taking more and more memory. And if you are opening more Modules as time goes on then memory is high on my suspect list.

Re: DXL Script Performance
Lee_Bost - Thu Feb 19 16:41:23 EST 2009

Doug.Zawacki - Thu Feb 19 16:35:08 EST 2009
I will take a look at the script. But I do have another question. When you are watching the CPU, how much memory are you using up? How much memory is on the workstation the script is running? Are you swapping memory out because of low memory? All these things can contribute to the issue. Of course, DXL Slowing down the longer it runs usually means it's taking more and more memory. And if you are opening more Modules as time goes on then memory is high on my suspect list.

The machine has 4gb of memory. As the script runs, doors.exe starts out using around 64mb. By the end it is using around 240mb. I have tried closing the modules after I have finished loading all of its objects before moving on to the next module but this doesn't improve the performance much.

Re: DXL Script Performance
Lee_Bost - Mon Feb 23 10:52:07 EST 2009

I did some testing and I notice that if I go back to the original speadsheet import in DOORS I have the same problem with it slowing down. I am at a complete lose on what it happening.

Re: DXL Script Performance
Doug.Zawacki - Mon Feb 23 11:12:00 EST 2009

Lee,

I have looked at the code. I see some places where you do not delete your temp skip lists. But, that will only add memory and you seem to have plenty of that available.
Therefore, If you'd like to try and dig into this script I will be happy to provide you with some performance monitor code that I have written.

After using the performance monitor code you should have an idea as to which functions take the longest amount of time but you may still not be able to find the answer. In fact, the answer may just be the interpreter itself. But you may find the results interesting and may learn something interesting.

Let me know if you are interested in using the performance monitor code. You will have to add calls at the top and bottom of each function you are interested in monitoring.

Re: DXL Script Performance
Lee_Bost - Mon Feb 23 11:43:04 EST 2009

Doug.Zawacki - Mon Feb 23 11:12:00 EST 2009
Lee,

I have looked at the code. I see some places where you do not delete your temp skip lists. But, that will only add memory and you seem to have plenty of that available.
Therefore, If you'd like to try and dig into this script I will be happy to provide you with some performance monitor code that I have written.

After using the performance monitor code you should have an idea as to which functions take the longest amount of time but you may still not be able to find the answer. In fact, the answer may just be the interpreter itself. But you may find the results interesting and may learn something interesting.

Let me know if you are interested in using the performance monitor code. You will have to add calls at the top and bottom of each function you are interested in monitoring.

That sounds good. I am willing to try anything at this point. Just to see what happens I ran a test over the weekend of the full data file which contains around 110,000 line. I started when I left work Friday and it was still running this morning. If I can narrow it down to where the problem is then maybe I will get lucky. Thanks for you help.

Re: DXL Script Performance
Doug.Zawacki - Mon Feb 23 12:15:34 EST 2009

Lee_Bost - Mon Feb 23 11:43:04 EST 2009
That sounds good. I am willing to try anything at this point. Just to see what happens I ran a test over the weekend of the full data file which contains around 110,000 line. I started when I left work Friday and it was still running this morning. If I can narrow it down to where the problem is then maybe I will get lucky. Thanks for you help.

OK...

Here is an example of how to use the performance monitor program.
Take the attached file (PerformanceMonitor.inc) and place it in the folder you want.
Notice, the code below is including it in c:\temp.
Place the PerfRegisterStart and PerfRegisterStop calls in any functions you want to measure. Run the code.

The output of the code will produce 2 files in the format:
PerfMon-Totals-YYYY-MM-DD@HH;MM;SS.csv
PerfMon-Details-YYYY-MM-DD@HH;MM;SS.csv

NOTE: Make sure to call PerfMonStop() to close the log files.

The example below should take about a minute to run. After that look at the two log files and I think you will see how this stuff all works. I really like using this performance monitor, it has helped answer a lot of questions for me.

Let me know if you have questions.
//////////////////////////////////////////////////////////////////////
pragma runLim, 0

#include <c:\temp\PerformanceMonitor.inc>

// Starts Performance Monitor
PerfMonStart()

void test1()
{
PerfRegisterStart("Test1")

int i,j
for(i=0;i<100;i++)
{
Skip sk = createString
for(j=0;j<1000;j++)
{
put(sk,"String" j "","string")
}
delete sk
}
PerfRegisterStop("Test1")
}

void test2()
{
PerfRegisterStart("Test2")
int i,j
for(i=0;i<1000;i++)
{
Skip sk = createString

for(j=0;j<1000;j++)
{
put(sk,"String" j "","string")
}
delete sk
}
PerfRegisterStop("Test2")
}
void test3()
{
PerfRegisterStart("Test3")

int i,j
for(i=0;i<10000;i++)
{
Skip sk = createString

for(j=0;j<1000;j++)
{
put(sk,"String" j "","string")
}
delete sk
}
PerfRegisterStop("Test3")
}

// Test
test1()
test2()
test3()

// MUST CALL PerfMonStop...this will close the log files !!!!
PerfMonStop()
print "done\n"
Attachments

attachment_14220368_PerformanceMonitor.inc

Re: DXL Script Performance
dpechacek - Mon Feb 23 12:39:11 EST 2009

Lee_Bost - Mon Feb 23 11:43:04 EST 2009
That sounds good. I am willing to try anything at this point. Just to see what happens I ran a test over the weekend of the full data file which contains around 110,000 line. I started when I left work Friday and it was still running this morning. If I can narrow it down to where the problem is then maybe I will get lucky. Thanks for you help.

If you're trying to import 110,000 lines with a comma separated file, you're probably just filling up the string table. Change any strings you can to Buffers. And be sure to delete them when proper and set them to null.

I wouldn't think it'd take an entire weekend though.

AAI Services, Textron
dpechacek@sc-aaicorp.com
David.Pechacek@gmail.com

Re: DXL Script Performance
Lee_Bost - Mon Feb 23 16:11:32 EST 2009

dpechacek - Mon Feb 23 12:39:11 EST 2009
If you're trying to import 110,000 lines with a comma separated file, you're probably just filling up the string table. Change any strings you can to Buffers. And be sure to delete them when proper and set them to null.

I wouldn't think it'd take an entire weekend though.

AAI Services, Textron
dpechacek@sc-aaicorp.com
David.Pechacek@gmail.com

The performance monitor was a great help. That is an awesome tool. With it I was able to find a couple of functions where I was creating and then deleting buffers in the function. One of the functions was call around 80000 times on my small 3300 record test file. By changing so that the buffer was only created and deleted one made a big improvement in the amount of time it was taking. When I try and run it against the large 110,000 line file it is better but it still needs some work. I think I am on the right track.

To try and improve the performance I have been trying to replace the strings with buffers. Especially in the part that is reading in the lines. But most of the time when I make a change the performance is worse. Do you know what would make the performance worse for buffers than for strings?

Re: DXL Script Performance
Doug.Zawacki - Mon Feb 23 16:44:09 EST 2009

Lee_Bost - Mon Feb 23 16:11:32 EST 2009
The performance monitor was a great help. That is an awesome tool. With it I was able to find a couple of functions where I was creating and then deleting buffers in the function. One of the functions was call around 80000 times on my small 3300 record test file. By changing so that the buffer was only created and deleted one made a big improvement in the amount of time it was taking. When I try and run it against the large 110,000 line file it is better but it still needs some work. I think I am on the right track.

To try and improve the performance I have been trying to replace the strings with buffers. Especially in the part that is reading in the lines. But most of the time when I make a change the performance is worse. Do you know what would make the performance worse for buffers than for strings?

Creating and deleting buffer does seems to make performance worse. If you can just create a global buffer to use I think you'll find that useful. As far as strings are concerened. I don't believe that you are having a string concatenation issue. That issue just takes up a lot of memory and does not affect performance as I believe you are seeing.

Focusing on the performance logs can help you isolate the functions that need to be streamlined. You should also spot any trends in which a function used to take 300 milliseconds to execute then takes 6 seconds at the end of the run. Thus, the reason for the detailed log (to spot trends) :)

Re: DXL Script Performance
Lee_Bost - Tue Feb 24 15:59:48 EST 2009

Doug.Zawacki - Mon Feb 23 16:44:09 EST 2009
Creating and deleting buffer does seems to make performance worse. If you can just create a global buffer to use I think you'll find that useful. As far as strings are concerened. I don't believe that you are having a string concatenation issue. That issue just takes up a lot of memory and does not affect performance as I believe you are seeing.

Focusing on the performance logs can help you isolate the functions that need to be streamlined. You should also spot any trends in which a function used to take 300 milliseconds to execute then takes 6 seconds at the end of the run. Thus, the reason for the detailed log (to spot trends) :)

Using the performance monitor I have fix a few problems that were slowing things down. It seems now I am down to the function below that is using the most time especially since it is called the most. Do you have any suggestions on how I could improve this function so that it doesn't take as long.

string impRemoveDoubleQuotes(Buffer buff)
{
string result = null
int i = 0
int buff_len = length buff
bool found = false

setempty(tempBuffer)
if (buff_len == 0)
return ""

if (buff[0] == impDoubleQuote)
i++

while (i < buff_len)
{
if (buff[i] == impDoubleQuote)
{
if (found)
{
found = false
}
else
{
tempBuffer += buff[i]
found = true
}
}
else
{
tempBuffer += buff[i]
}
i++
}

buff_len = length tempBuffer
if (tempBufferbuff_len - 1 == impDoubleQuote)
result = tempBuffer0:buff_len - 2 ""
else
result = tempBuffer0:buff_len - 3 ""

return result

} /* impRemoveDoubleQuotes */

Re: DXL Script Performance
Doug.Zawacki - Tue Feb 24 19:04:11 EST 2009

Lee_Bost - Tue Feb 24 15:59:48 EST 2009
Using the performance monitor I have fix a few problems that were slowing things down. It seems now I am down to the function below that is using the most time especially since it is called the most. Do you have any suggestions on how I could improve this function so that it doesn't take as long.

string impRemoveDoubleQuotes(Buffer buff)
{
string result = null
int i = 0
int buff_len = length buff
bool found = false

setempty(tempBuffer)
if (buff_len == 0)
return ""

if (buff[0] == impDoubleQuote)
i++

while (i < buff_len)
{
if (buff[i] == impDoubleQuote)
{
if (found)
{
found = false
}
else
{
tempBuffer += buff[i]
found = true
}
}
else
{
tempBuffer += buff[i]
}
i++
}

buff_len = length tempBuffer
if (tempBufferbuff_len - 1 == impDoubleQuote)
result = tempBuffer0:buff_len - 2 ""
else
result = tempBuffer0:buff_len - 3 ""

return result

} /* impRemoveDoubleQuotes */

Hi Lee,

In just a quick look I have one suggestion.

// Current code
setempty(tempBuffer)
if (buff_len == 0)
return ""

// Suggested Change
if (buff_len == 0)
return ""
setempty(tempBuffer)

I will take a look at the function closer when I have more time. BTW, what are the stats on this function? How many times called? Min. Time to execute? Max. time to execute?

Re: DXL Script Performance
Lee_Bost - Wed Feb 25 12:28:57 EST 2009

Doug.Zawacki - Tue Feb 24 19:04:11 EST 2009
Hi Lee,

In just a quick look I have one suggestion.

// Current code
setempty(tempBuffer)
if (buff_len == 0)
return ""

// Suggested Change
if (buff_len == 0)
return ""
setempty(tempBuffer)

I will take a look at the function closer when I have more time. BTW, what are the stats on this function? How many times called? Min. Time to execute? Max. time to execute?

On a 10,000 record file the function is call around 150,000 times. At the begining execution time is 0, by the end it is taking around 15 to 16 milliseconds. Every once in awhile there will be an execution that takes around 31 milliseconds. The thing I don't understand is why at the begining it is running so well and then slows down. The other weird thing is that I will see times in the execution that it will speed back up for a little while and then slow back down. Would the sheer number of calls to the function cause it to slow down? Would it help since this function is called in only one place to make it part of the calling function?

Re: DXL Script Performance
Doug.Zawacki - Wed Feb 25 15:27:37 EST 2009

Lee_Bost - Wed Feb 25 12:28:57 EST 2009
On a 10,000 record file the function is call around 150,000 times. At the begining execution time is 0, by the end it is taking around 15 to 16 milliseconds. Every once in awhile there will be an execution that takes around 31 milliseconds. The thing I don't understand is why at the begining it is running so well and then slows down. The other weird thing is that I will see times in the execution that it will speed back up for a little while and then slow back down. Would the sheer number of calls to the function cause it to slow down? Would it help since this function is called in only one place to make it part of the calling function?

Lee,

The minimum resolution of the Performance Monitor is 15 milliseconds. This is due to a limitation of DOORs and not the monitor itself. Therefore, it seems that the function is most likely taking less than 15 milliseconds to run. The times you see it hit a 15 mark is probably at the time DOORS updates it's tick timer.

Please try the change I suggested above to see if that helps any. (moving the length check above the setempty)....

I should mention that you may be seeing performance degradation due to the amount of history you may be generating in the modules themself. Attribute history and possibly link history (if you have history enabled for those things). Also, you can try a baseline and purge to the modules prior to running the updates.

Lastly, I have seen a strange but serious performance hit when dealing with substring references on strings larger than 60,0000 characters. eg. mystring3: . Not sure if you are encountering the same within this function or even when utilizing buffers. I have a hard time believing this to be the case but I thought I should mention it.

Keep us posted if you find anything interesting.

Re: DXL Script Performance
Lee_Bost - Fri Feb 27 11:20:36 EST 2009

Doug.Zawacki - Wed Feb 25 15:27:37 EST 2009
Lee,

The minimum resolution of the Performance Monitor is 15 milliseconds. This is due to a limitation of DOORs and not the monitor itself. Therefore, it seems that the function is most likely taking less than 15 milliseconds to run. The times you see it hit a 15 mark is probably at the time DOORS updates it's tick timer.

Please try the change I suggested above to see if that helps any. (moving the length check above the setempty)....

I should mention that you may be seeing performance degradation due to the amount of history you may be generating in the modules themself. Attribute history and possibly link history (if you have history enabled for those things). Also, you can try a baseline and purge to the modules prior to running the updates.

Lastly, I have seen a strange but serious performance hit when dealing with substring references on strings larger than 60,0000 characters. eg. mystring3: . Not sure if you are encountering the same within this function or even when utilizing buffers. I have a hard time believing this to be the case but I thought I should mention it.

Keep us posted if you find anything interesting.

Well, I think I finally have the script running in an acceptible amount of time. It is still taking longer than I would like but I will keep looking for ways to improve it. For now I want to share what changes have made the most improvements.

1) Create and delete buffers as little as possible. In my case I had a function that was creating and delete a buffer each time it was run (around 80,000 times). By creating and deleting the buffer once globally my script went from running for 2 hours to 20 minutes.

2) Create and delete skip list as little as possible. My script runs faster when I delete each item in the skip list before using it again instead of deleting the entire skip list and then recreating it.

3) Try to eliminate writing to buffers one character at a time. I ended up rewriting the impRemoveDoubleQuotes function to search for the double quotes and then write chunks to the tempBuffer instead of write each character. This has improve the speed by about 35%. The updated function is included below.

So far it has been my experience that buffers, while not taking up space in the string table, are not the most efficient. Thanks for all the help.

Lee

string impRemoveDoubleQuotes(Buffer buff)
{
int curDQLoc = 0
int buff_len = length buff
int searchStart = 1

if (buff_len <= 3)
return ""
setempty(tempBuffer)

curDQLoc = contains(buff, impDoubleQuote, searchStart)
while ((curDQLoc != -1) && (searchStart != buff_len))
{
if (curDQLoc == buff_len-2)
{
if (searchStart <= curDQLoc-1)
{
tempBuffer += buffsearchStart:curDQLoc-1
}
searchStart = curDQLoc + 2
}
else
{
if (buffcurDQLoc+1 == impDoubleQuote)
{
tempBuffer += buffsearchStart:curDQLoc
searchStart = curDQLoc + 2
}
else
{
tempBuffer += buffsearchStart:curDQLoc
searchStart = curDQLoc + 1
}
}
curDQLoc = contains(buff, impDoubleQuote, searchStart)
}
if (searchStart < buff_len-1)
{
tempBuffer += buffsearchStart:buff_len-1
}
return tempBuffer0:

} /* impRemoveDoubleQuotes */

Re: DXL Script Performance
Lee_Bost - Tue Mar 03 11:10:07 EST 2009

Lee_Bost - Fri Feb 27 11:20:36 EST 2009
Well, I think I finally have the script running in an acceptible amount of time. It is still taking longer than I would like but I will keep looking for ways to improve it. For now I want to share what changes have made the most improvements.

1) Create and delete buffers as little as possible. In my case I had a function that was creating and delete a buffer each time it was run (around 80,000 times). By creating and deleting the buffer once globally my script went from running for 2 hours to 20 minutes.

2) Create and delete skip list as little as possible. My script runs faster when I delete each item in the skip list before using it again instead of deleting the entire skip list and then recreating it.

3) Try to eliminate writing to buffers one character at a time. I ended up rewriting the impRemoveDoubleQuotes function to search for the double quotes and then write chunks to the tempBuffer instead of write each character. This has improve the speed by about 35%. The updated function is included below.

So far it has been my experience that buffers, while not taking up space in the string table, are not the most efficient. Thanks for all the help.

Lee

string impRemoveDoubleQuotes(Buffer buff)
{
int curDQLoc = 0
int buff_len = length buff
int searchStart = 1

if (buff_len <= 3)
return ""
setempty(tempBuffer)

curDQLoc = contains(buff, impDoubleQuote, searchStart)
while ((curDQLoc != -1) && (searchStart != buff_len))
{
if (curDQLoc == buff_len-2)
{
if (searchStart <= curDQLoc-1)
{
tempBuffer += buffsearchStart:curDQLoc-1
}
searchStart = curDQLoc + 2
}
else
{
if (buffcurDQLoc+1 == impDoubleQuote)
{
tempBuffer += buffsearchStart:curDQLoc
searchStart = curDQLoc + 2
}
else
{
tempBuffer += buffsearchStart:curDQLoc
searchStart = curDQLoc + 1
}
}
curDQLoc = contains(buff, impDoubleQuote, searchStart)
}
if (searchStart < buff_len-1)
{
tempBuffer += buffsearchStart:buff_len-1
}
return tempBuffer0:

} /* impRemoveDoubleQuotes */

I have the answer. On a whim, I changed tempBuffer from a buffer to a string in the impRemoveDoubleQuotes function that I posted above. Making this change caused the script to go from running in 7.5 hours to just under 4 minutes. I was concerned making this change was use more memory. As it turns out it is using 300MB less. I don't understand why this change seems to work but at least the script is running in a reasonable amount of time. Maybe is has to do with the number of times (at least 80,000) data is written to the buffer. Does any one have thoughts on why this change makes a difference?